RU-EVAL-2012: Evaluating Dependency Parsers for Russian
نویسندگان
چکیده
The paper reports on the recent forum RU-EVAL ‒ a new initiative for evaluation of Russian NLP resources, methods and toolkits. It started in 2010 with evaluation of morphological parsers, and the second event RU-EVAL 2012 (2011-2012) focused on syntactic parsing. Eight participating IT companies and academic institutions submitted their results for corpus parsing. We discuss the results of this evaluation and describe the so-called “soПt” evaluation principles that allowed us to compare output dependency trees, which varied greatly depending on theoretical approaches, parsing methods, tag sets, and dependency orientations principles, adopted by the participants. TITLE AND ABSTRACT IN RUSSIAN RU-EVAL-2012: Оц а а а а а я я ы а RU-EVAL – , . 2010 (LвКsСОvskКвК Оt Кl. 2010), (20112012) (TolНovК Оt Кl. 2012). 8 . « » , , , . , , , .
منابع مشابه
Cross-Framework Evaluation for Statistical Parsing
A serious bottleneck of comparative parser evaluation is the fact that different parsers subscribe to different formal frameworks and theoretical assumptions. Converting outputs from one framework to another is less than optimal as it easily introduces noise into the process. Here we present a principled protocol for evaluating parsing results across frameworks based on function trees, tree gen...
متن کاملParsing Any Domain English text to CoNLL dependencies
It is well known that accuracies of statistical parsers trained over Penn treebank on test sets drawn from the same corpus tend to be overestimates of their actual parsing performance. This gives rise to the need for evaluation of parsing performance on corpora from different domains. Evaluating multiple parsers on test sets from different domains can give a detailed picture about the relative ...
متن کاملHybrid Combination of Constituency and Dependency Trees into an Ensemble Dependency Parser
Dependency parsing has made many advancements in recent years, in particular for English. There are a few dependency parsers that achieve comparable accuracy scores with each other but with very different types of errors. This paper examines creating a new dependency structure through ensemble learning using a hybrid of the outputs of various parsers. We combine all tree outputs into a weighted...
متن کاملA Dependency-based Method for Evaluating Broad-Coverage Parsers
A b s t r a c t Wi th th t emergence of broad-coverage parsers, quan titative evaluation of parsers becomes increasingly more important We propose a dependency-based method for evaluating broad-coverage parsers The method offers several advantages over previous methods that are based on phrase boundaries The error count score WL propose here is not only more mtui t ivt lv meaningful than other...
متن کاملLearning Computational Linguistics through NLP Evaluation Events: the experience of Russian evaluation initiative
We present in the paper our experience of involving the students of the department of theoretical and computational linguistics of the Moscow State University into full-cycle activities of preparing and evaluating the results of the NLP Evaluation forums, held in 2010 and 2012 in Russia. The forum of 2010 started as a new initiative and was the first independent evaluation of morphology parsers...
متن کامل